Method and Apparatus for Filling In or Replacing Image Pixel Data

ABSTRACT

A system for filling in and/or replacing pixel data in a target image uses pixel data from a source image. In one embodiment, the pixel data in the source image are classified and boundaries of local class areas or groups of similarly classified pixels are determined. The pixel data in the local class areas are compared to determine one or more scaling factors. The missing pixel data or data to be replaced in the target image is obtained from the source image and scaled with the one or more scaling factors.

FIELD

The technology disclosed herein relates to image processing techniques.

BACKGROUND

In the field of remote sensing, aerial and/or satellite images ofterrain are studied to determine information about the land that isimaged. For example, in the forest products industry, aerial orsatellite images of timberlands are used because the lands are often toovast or remote to access and survey on foot. The images of thetimberlands may be analyzed to determine such information as theboundaries of a forest, the age of the trees in the forest, the types oftrees in the forest, and a variety of other information.

In some cases, digital, aerial, or satellite images of terrain willcontain missing pixel data. For example, a well-publicized mechanicalfailure on the Landsat 7 satellite causes images to be produced withbands of missing pixel data. Similarly, the terrain depicted in anaerial or satellite image may be fully or partially obscured by clouds,shadows, or other phenomena, thereby making it difficult to analyze.

Most commonly, images with missing pixel data or data that does not showwhat is desired are corrected with pixel duplication techniques wherebythe pixel data to be filled in or replaced are obtained from the imageitself and are assumed to be similar to the data for neighboring goodpixels. However, such techniques often produce poorly-corrected images.Therefore, there is a need for a technique that can fill in missingpixel data and/or replace pixel data that produces a betterapproximation of how the underlying object or terrain in an imageactually appears.

SUMMARY

To address the problems discussed above, the technology disclosed hereinrelates to a method and apparatus for modifying pixel data by filling inmissing pixel data and/or replacing pixel data in an image. Inaccordance with one embodiment, a target image with missing pixel dataor pixel data representing an obscured object is replaced with pixeldata from a source image showing the same object or terrain. The sourceimage may be obtained at a different time than the target image and maybe obtained with a different imaging system. The source image and targetimage are aligned and the source image is re-sampled, if required, tohave the same resolution as the target image. Pixel data from the sourceimage are classified into groups and the boundaries of local areas ofsimilarly classified pixels are determined. Pixel data in the targetimage are filled in and/or replaced by the corresponding pixel data inthe source image that are adjusted by one or more scaling factors.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This summary is not intended to identify key features ofthe claimed subject matter, nor is it intended to be used as an aid indetermining the scope of the claimed subject matter.

DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many of the attendant advantages of thedisclosed technology will become more readily appreciated as the samebecome better understood by reference to the following detaileddescription, when taken in conjunction with the accompanying drawings,wherein:

FIG. 1 diagrammatically illustrates a target Landsat 7 image withmissing pixel data;

FIG. 2A represents a portion of the target image having valid andmissing pixel data;

FIG. 2B represents a portion of a source image having valid pixel data;

FIG. 2C represents the portion of the target image shown in FIG. 2A thatis filled with pixel data from the source image;

FIGS. 3A and 3B illustrate an example of how pixel data in a targetimage are filled and/or replaced in accordance with one embodiment ofthe disclosed technology;

FIGS. 4A-4C illustrate a series of steps performed to fill in and/orreplace pixel data in a target image in accordance with one embodimentof the disclosed technology; FIG. 5 illustrates a representative systemfor filling in and/or replacing pixel data in a target image inaccordance with an embodiment of the disclosed technology;

FIG. 6A illustrates a portion of an actual target image that is missingpixel data;

FIG. 6B illustrates a portion of an actual source image with valid data;and

FIG. 6C illustrates the target image of FIG. 6A with data filled in fromthe source image.

DETAILED DESCRIPTION

As indicated above, the disclosed technology relates to a method andapparatus for filling in missing pixel data or replacing pixel data in atarget image. Although the technology is described with respect toaerial and satellite images of terrain, it will be appreciated that thedisclosed technology can be used to fill in missing pixel data and/orreplace pixel data in any digital image.

FIG. 1 illustrates a representative target image 50 that contains validpixel data in a number of areas 52 and one or snore gaps 54 that aremissing pixel data. In the case of Landsat 7 satellite images, amechanical fault on the satellite produces images of terrain thatcontain gaps 54 of missing pixel data. In order to fill in the gaps inthe target image, the disclosed technology replaces the missing pixeldata with pixel data that are obtained from a source image of the sameterrain.

FIG. 2A shows a portion 60 of the target Landsat 7 image 50 having anumber of pixels 68 with valid data, and a number pixels that aremissing data as indicated by the question marks. To fill in the pixeldata from corresponding pixels 72 in a source image 70 (shown in FIG.2B) are used.

FIG. 2C shows the portion 60 of the target Landsat 7 image including theoriginal pixels 68 with valid pixel data and pixels 72 with data thatare obtained from the source image 70. With the missing pixel data addedto the target image, the target image can be analyzed for its desiredpurpose.

The source image from which the missing pixel data are obtained may becaptured at or about the same time that the target image is capturedusing a different imaging system. Alternatively, the source image may becaptured at an earlier time than when the target image is captured usingeither the same or a different imaging system than the one used toobtain the target image. In most cases, the source image will becaptured earlier than the target image, such that the target imagerepresents the most recent image of the terrain in question. However, itis possible that a more recent image could be used as a source image tofill in missing pixel data or to replace pixel data in apreviously-captured target image.

To fill in the missing pixel data or to replace pixel data in the targetimage, the pixel data of a source image are classified. Digitalsatellite and aerial image data typically contain values for each pixelin multiple spectral bands including one or more visible spectral bands,e.g., red, green, blue, and one or more infrared bands, e.g., nearinfrared IR₁, medium infrared IR₂, and far infrared IR₃. If an 8-bitimaging system is used, then the pixel value for each band can vary from0-255. If there are 6 spectral bands, then the total number of differentpixel data combinations is 256⁶. Classification serves to reduce thenumber of pixel combinations by identifying pixels in an image havingsimilar spectral values. In one embodiment, a classifying engine, suchas ERDAS from the Leica Corporation, can be used to classify pixel datainto either a predefined number of classes or a varying number ofclasses where the pixel data values vary by no more than auser-specified amount from each other. The details of how to classifydata are considered to be known to those of ordinary skill in the artand are therefore not discussed in further detail.

In one embodiment, the pixel data are classified into approximately 120different classes. The pixels assigned to each class can bestatistically analyzed to determine, for example, the mean or medianvalue of the data in any spectral band, etc. The classifying engine alsodetermines the boundaries of local areas or groupings of pixels havingthe same classification. Typically, if a classified pixel does not havea sufficient number of neighbors (e.g., <50) with the sameclassification, the pixel is re-classified.

One embodiment of a technique for filling in missing pixel data in atarget image in accordance with the disclosed technology is illustratedin FIGS. 3A and 3B. In the example shown, a target image 100 representsan urban back yard having areas of grass and a swimming pool. In theexample shown, one area of grass is missing data for a single pixel 102,while a second area of grass is missing data for two pixels 104, 106.The target image 100 is also missing all data for the pixels 110 in thearea of the swimming pool.

FIG. 3B illustrates a source image 120 of the same terrain that may havebeen obtained at a different time or with a different imager. The sourceimage and the target image are aligned so that a pixel in the sourceimage represents the same physical location as the corresponding pixelin the target image. In addition, the source image and the target imagepreferably have the same resolution. If the source and target images donot have the same resolution, then the source image may need to bere-sampled, as appropriate.

Each pixel in the source image is classified and the boundaries of localareas or groups of pixels with the same classification are determined.In the example shown, the source image has a first local class area 150that represents the first grass patch and a second local class area 152that represents the second grass patch. A local class area 1 60 includespixels in the area of the swimming pool and a local class area 162 haspixels showing a neighboring swimming pool.

To fill in data for the missing pixels, such as pixel 102, thecorresponding replacement pixel is found in the source image. To accountfor differences in the type of camera used to obtain the target andsource image, the spectral bands of the source and target image arecorrelated. For example, the pixel data for the target image may havevalues for seven spectral bands, while the pixel data for the sourceimage may only have values for four spectral bands. Therefore, the pixeldata in the source and target images are correlated to determine whichspectral band in one of the images corresponds the best to whichspectral band in the other image. The data values for the spectral bandhaving the best correlation coefficient in the source image are used tofill in or replace the data of the correlated spectral band in thetarget image.

To account for differences in the reflectance of the image that mayoccur between the time at which the source image is captured and whenthe target image is captured (e.g., the source image and target imagemay be captured at different times of year or under different lightingor weather conditions), a scaling factor representing a radiometricrelationship between each spectral band to be Filled in or replaced isdetermined. In one embodiment, the scaling factors are computed bycomparing pixel data in corresponding local class areas in the targetimage and source image. For example, the local class area 150 representsa grass patch in the source image 120. The boundaries of the local classarea 150 in the source image define the corresponding boundaries of agroup of pixels in the target image with which the scaling factors canbe computed. For example, the scaling factors may be determined bycomparing the mean values for pixel data in each spectral band in theboundaries of the local class area within the source and target images.

In the example shown, the mean values in each spectral band for thepixels that are within the local class area 150 that surrounds the pixel102 are determined. The mean values of the data for each spectral bandfor the pixels in the local area 150 in the source image are alsodetermined, and the mean values are compared. In one embodiment, theratio of the means for each spectral band is used as the scaling factor.

As an alternative to using the ratio of the means, a regression equationmay be built using corresponding pixels from both source and targetimages to define the radiometric relationship. The missing pixel datavalues for the target image can be predicted using this equation.

To fill in the missing data for the pixel 102, the data for thecorresponding pixel in the source image is obtained and is scaled by thescaling factors. In the example shown, the data for the pixel in thesource image that corresponds to the pixel 102 are obtained. The datafor one spectral band, such as the red spectral band, are computed bytaking the red spectral band data (or a different spectral band, if thedifferent band correlates better) and multiplying it by the red spectralband scaling factor. Data for the other spectral bands of the pixel 102are obtained and scaled in the same way.

In the example shown, the source image 120 may be taken in the earlyspring when the grass is dormant, while the target image 100 is taken inthe summer. By scaling the spectral data by the scaling factors, thepixel data from the source image will more closely match the data forthe pixels that surround pixel 102.

In some instances, there aren't enough valid pixels in a local classarea of the target image to compute the scaling factors. If more than auser determined percentage (e.g., >90%) of pixels in a local class areaare missing data or need to have the data replaced, then one embodimentof the disclosed technology uses a global class comparison to computethe scaling factors.

In the example of FIG. 3A, the data for the pixels 110 representing theswimming pool in the target image are all missing. However, theclassifier has identified a local class area 160 that corresponds to theswimming pool in the target image. In addition, a local class area 162has similarly classified pixels since the water in most swimming poolswill generally look the same from the air. The boundaries of the localclass area 162 in the source image define boundaries of a correspondingarea 162 that contain a group of valid pixels in the target image.Therefore, the pixel data in the corresponding area 162 in the targetimage can be analyzed and compared with the pixel data for the pixels inthe local class areas 160 and 162 in the source image to compute thescaling factors. If a local class area is relatively small, then ameaningful statistical comparison might not be accurate. In this case,the scaling factors can be determined by comparing the pixel data forall the similarly classified pixels in the source image with those incorresponding positions in the target image (a global class comparison).The data for the missing pixels 110 in the target image are determinedby taking the spectral data for the corresponding pixels in the sourceimage that are included in boundaries of the local class area 160 andscaling the data by the determined scaling factors.

FIGS. 4A and 4C illustrate a series of steps performed to fill inmissing pixel data and/or to replace pixel data for obscured portions ofan image in accordance with one embodiment of the disclosed technology.Although the steps are set forth in a particular order for ease ofdiscussion, it will be appreciated that the order may be changed ordifferent steps performed in order to achieve the functionalitydescribed.

Beginning at 200, a target image that is missing pixel data or that hasdata to be replaced is selected. At 202, a source image covering thesame area of interest (e.g., parcel of terrain) is selected. At 204, thetarget image and source image are registered so that pixels at the sameposition in the target and source images correspond to the samegeographic location, in the case of images of terrain, or the sameobject or portion thereof, in the case of images of other things. Inaddition, the source image may be re-sampled to have the same resolutionas the target image. The re-sampling may occur prior to the registrationof the source and target image. Alternatively, the re-sampling may bedone during the data filling operation. At 206, the correspondingspectral bands of the target image and the source image are determinedbased on calculated band correlation coefficients. At 208, the pixeldata in the source image is classified, and the boundaries of the localclass areas are determined.

Turning to FIG. 4B, a minimum mapping unit (MMU) or minimum size localarea class is determined. The MMU may be determined as a percentage ofthe size of the source or target images or may be user selected. A localclass area with a size that is less than the MMU is merged with aneighboring local class area at 210.

At 212, a loop begins processing each pixel in the target image to befilled or to have its data replaced.

At 214, a corresponding pixel in a source image is determined, includingthe source image pixel's class and the boundaries of its local classarea. At 216, it is determined if there are a sufficient number ofpixels to analyze in the corresponding local class area in the targetimage. If a local class area comparison can be made, the data for thepixels in the local class area in the target image are compared to thedata for the pixels in the local class area in the source image in orderto determine the scaling factors.

If there are not enough valid pixels present in the local class area inthe target image or if the local class area is too small, then a globalclass comparison of similarly classified pixels in the source and targetimages is made to determine the scaling factors.

In FIG. 4C, the missing pixel data is filled in or replaced with thepixel data from the corresponding pixel in the source image afterscaling by the scaling factors at 222. If the source image and thetarget image have different resolutions, then re-sampling of the sourceimage may be performed by, for example, a bi-linear transformation, todetermine the data for the target pixel.

At 224, it is determined if all the missing pixel data have been filledin or if all the data for the identified pixels has been replaced. Ifso, processing quits. If not, processing returns to step 212 until allthe gaps or missing pixel data have been filled in or all the desiredpixel data have been replaced.

FIG. 5 illustrates one representative computer system that can be usedto implement the disclosed technology. A computer 300 includes one ormore processors that execute a sequence of programmed instructions thatcause the computer to fill in missing pixel data and/or to replace pixeldata in a target image in accordance with the techniques describedabove. The instructions for the processor(s) may be stored on a computerstorage media 302 such as a CD ROM, flash drive, hard drive, etc., or aremote computer (not shown) that is in communication with the computer300 via a wired or wireless connection to a communication network suchas the Internet 310. The computer 300 receives the source and targetimages from a computer storage media, camera memory card, or over acommunication network and fills in missing pixel data and/or replacespixel data as described. The processed images are displayed for a useron a video monitor or other display 312 and may be stored or transmittedto a remote location for analysis.

FIG. 6A illustrates a portion of an actual target satellite image 400 ofterrain having an area of valid pixel data and that is missing pixeldata in band gaps 410 and 420. FIG. 6B illustrates a portion of anactual source image 430 of the same terrain that was captured with adifferent satellite system. FIG. 6C shows the same target image 400 ofFIG. 6A with pixel data obtained from the source image 430 are used tofill the band gaps.

While illustrative embodiments have been illustrated and described, itwill be appreciated that various changes can be made therein withoutdeparting from the scope of the invention as defined by the followingclaims and equivalents thereof.

1. A method of operating a computer system to modify pixel data in atarget image, comprising: storing a target image of an object withpixels to be modified in a memory of the computer; storing a sourceimage of the same object in a memory of the computer; classifying pixeldata in the source image to identify pixels having similar data valuesand determining boundaries of local class areas containing pixels thatare grouped in a same class; determining a pixel in the target image tobe modified; identifying a replacement pixel in the source image havingdata that will replace the data for the pixel in the target image to bemodified; comparing data values for pixels in a local class area in thesource image, that includes the replacement pixel, to the data valuesfor pixels in the target image to determine radiometric relationshipsbetween the pixel data in the source image and the pixel data in thetarget image; and modifying the pixel in the target image by replacingthe data with the data for the replacement pixel using the determinedradiometric relationships.
 2. The method of claim 1, wherein the datavalues for pixels in the source image and in the target image arecompared by: comparing data values for pixels in the target image thatare within a boundary that corresponds to the boundary of the localclass area in the source image that includes the replacement pixel. 3.The method of claim 1, wherein the data values for pixels in the sourceimage and in the target image are compared by: comparing data values forpixels in the target image that are in boundaries that correspond toboundaries of two or more local class areas in the source image thatcontain pixels that are in the same class as the replacement pixel. 4.The method of claim 2, wherein each pixel in the source image and thetarget image has data for multiple spectral bands and the radiometricrelationships are determined by comparing a mean value of the data foreach spectral band for the pixels in the target image with the meanvalue of the data for each spectral band for the pixels in the sourceimage.
 5. The method of claim 2, wherein each pixel in the source imageand the target image has data for multiple spectral bands and theradiometric relationships are determined by computing a regressionfunction that relates the values of the pixel data for each spectralband in the target image with the values of the pixel data for eachspectral band in the source image.
 6. The method of claim 3, whereineach pixel in the source image and the target image has data formultiple spectral bands and the radiometric relationships are determinedby comparing a mean value of the data for each spectral band for thepixels in the target image with the mean value of the data for eachspectral band for the pixels in the source image.
 7. The method of claim3, wherein each pixel in the source image and the target image has datafor multiple spectral bands and the radiometric relationships aredetermined by computing a regression function that relates the values ofthe pixel data for each spectral band in the target image with thevalues of the pixel data for each spectral band in the source image. 8.The method of claim 1, wherein the pixels in the target image have datafor a first set of spectral bands and the pixels in the target imagehave data for a second set of spectral bands and wherein the methodfurther comprises: comparing the data for pixels in the source image andthe target image to determine which spectral bands in the source imagecon-elate with which spectral bands in the target image, wherein thedata for the pixel to be modified in the target image is replaced withthe data for the spectral band for the replacement pixel that has a bestcorrelation.
 9. A computer storage media having stored thereon asequence of programmed instructions that are executable by a computersystem to perform any of method claims 1-8.
 10. A computer system tomodify pixel data in a target image, comprising: a memory that isconfigured to store a target image of an object with pixels to bemodified and a source image of the same object; a processor that isprogrammed to: classify pixel data in the source image to identifypixels having similar data values and to determine boundaries of localclass areas containing pixels that are grouped in a same class;determine a pixel in the target image to be modified; identify areplacement pixel in the source image having data that will replace thedata for the pixel in the target image to be modified; compare datavalues for pixels in a local class area in the source image thatincludes the replacement pixel to the data values for pixels in thetarget image to determine radiometric relationships between pixel datain the source image and pixel data in the target image; and modify thepixel in the target image by replacing the data with the data for thereplacement pixel using the determined radiometric relationships. 11.The computer system of claim 10, wherein the radiometric relationshipscomprise ratios of mean spectral values of pixels within a boundary ofthe local class area in the source image to mean spectral values ofpixels in the target image.
 12. The computer system of claim 10, whereinthe radiometric relationships are defined by regression functions thatrelate spectral data for pixels in the source image and spectral datafor pixels in the target image.
 13. A system for modifying pixel data ina target image, comprising: a memory configured to store a target imageand a source image are stored along with boundaries of pixels in thesource image having similar spectral values; a processor that isconfigured to execute a program to: determine one or more pixels in thetarget image having data in a number of spectral bands to be modified;compute a scaling factor for each spectral band that relates thespectral values of pixels in a boundary area in the source image tospectral values of pixels having data that are not to be modified in thetarget image; and modify data for a pixel representing a feature in thetarget image by replacing the spectral band data with spectral band datafor a corresponding pixel representing the same feature in the sourceimage scaled by the scaling factor computed for each spectral band.